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•  Results  of  the  Experimentation  Workshop 

-Workshop  Chair  Brief  to  the  MORS  Sponsors 

•  Experimentation  Community  of  Practice 

-  Background 

-  Organization 

-  Next  Steps 

•  Potential  WG  33  Name  Change 
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Bringing  Analytical  Rigor  to  Joint 
Warfighting  Experimentation: 

Design,  Planning,  Execution,  Analysis  and  Reporting 
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Sponsor  Brief 

Kirk  Michealson,  Workshop  Chair 


AGENDA 


Military  Operations  Research  Society 


•  Special  Meeting  Background 

•Key  Changes  &  Enduring  Challenges 

•  Overall  Summary 

-  Overarching  Objectives 

-  JFCOM  /  J-9  Analytical  Challenges 

-  Working  Group  Discussions 

•  Status  &  Final  Thoughts 


EXPERIMENTATION  WORKSHOP 

Background:  Workshop  Overview 
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Bringing  Analytical  Rigor  to 
Joint  Warfighting  Experimentation 

•  WHEN:  Monday-Thursday,  2-5  October  2006 

•  WHERE:  Joint  Forces  Staff  College,  Norfolk,  VA 

•  FORMAT: 

-  Tutorial,  Monday,  2  October  2006 

-  Mini-Symposium,  Tuesday,  3  October  2006 

-  Workshop,  Wednesday-Thursday,  4-5  October  2006 

•  ATTENDANCE 

-  Special  meeting:  113-61  members  and  52  non-members 

•  5  foreign:  Canada,  Australia  and  Germany 

-  Tutorial:  ~  40 


EXPERIMENTATION  WORKSHOP 

Background:  Workshop  Purpose  and  Goal 
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•  Workshop  Purpose: 

-  Provide  a  forum  for  discussing  approaches  to  warfighting 
experimentation 

-  Provide  an  opportunity  for  miiitary  and  civilian  operations 
research  analysts  to  examine  topics,  methodologies,  anaiyses, 
and  innovations  pertinent  to  aii  aspects  of  designing, 
executing,  analyzing,  and  reporting  joint  warfighting 
experiments. 

•  Workshop  Goal: 

-  The  goai  of  the  1 999  Joint  Experimentation  Workshop  was  to 
develop  an  experimentation  process 

-  Using  the  results  of  the  1999  Workshop  and  the  two  published 
Codes  of  Best  Practice  for  Experimentation,  the  goal  of  this 
workshop  is  to  determine  how  to  appiy  analyticai  rigor  across 
the  process. 
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EXPERIMENTATION  WORKSHOP 

Background:  Leadership 
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•  Workshop  Chairs 

-Kirk  Michealson,  Lockheed  Martin  Center  for  Innovation 

-  Dr.  Mike  Cochrane,  JFCOM  /  J-9 

•  Technical  Chair 

-Dr.  Richard  Hayes,  EBR,  Inc. 

•  Bulldog 

-Dr.  Lee  Lehmkuhl,  MITRE  Corp. 

•  Synthesis  Group  Chair 

-  Roy  Reiss,  FS,  HQ  AF/A9 
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EXPERIMENTATION  WORKSHOP 

Overview:  Working  Groups 
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•  WG  1:  Experiment  Design 

-  Co-Chairs:  Steve  Boothe,  COMOPTEVFOR  &  Dennis  DeRiggi,  IDA 

•  WG  2:  Data  Collection,  Metrics  Evaluation  &  Reporting 

-  Chair:  Rick  Rigazio,  NWDC 

•  WG3:  Methodologies  and  Tools 

-  Chair:  Scott  Hamilton,  ACC  AFC2ISRC/AFEO  (L-3  Comm) 

•  WG  4:  People  as  Experimental  Assets 

-  Chair:  Alex  Hoover,  JFCOM/JKDDC  (Sparta,  Inc.) 

•  WG  5:  Integration  &  Coordination  Across  DoD 

-  Chair:  Mike  McGinnis,  VMASC 

•  WG  6:  Analytical  Rigor  to  Support  Capabilities  Based  Planning 

-  Chair:  Teresa  Wilson,  Lockheed  Martin  Aerospace 


9 


EXPERIMENTATION  WORKSHOP 

Overview:  Mini-Symposium 
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•  Keynote  Presentation 

-  Mr.  Dave  Ozolek,  Deputy  JFCOM  /  J-9 

•  Joint  Presentations 

-  Col  Eileen  Bjorkman,  Director,  Joint 
Test  and  Evaluation  Project 

-  Mr.  Richard  Marchant,  Director,  Joint 
Experimentation  Project,  Office  of  the 
Assistant  Deputy  Secretary  of  Defense 
for  Joint  &  Coalition  Operations 
Support 

-  Mr.  Mel  Chaloupka,  Head  of 
Experimentation  and  Transformation 
at  USPACOM 

-  LTC  Bryan  Luke,  Chief  Joint 
Experimentation,  Transformation,  and 
Concepts  Division,  Joint  Staff  /  J-7 


•  Service  and  Coalition 
Presentations 

-  Col  Mike  Wilmer,  Studies  and 
Analysis  Chief,  Army  Capabilities 
Integration  Center 

-  CDR  Steve  Swittel,  Experiment 
Coordinator,  Naval  Warfare 
Development  Command 

-  Col  Larry  King,  Director,  Joint 
Concept  Development  and 
Experimentation,  MCCDC 

-  Mr.  Michael  Knollman,  Assistant 
Deputy  Under  Secretary  of  Defense 
for  Joint  and  Coalition  Operations 

-  Mr.  Michael  Wahl,  Chief  Analyst, 
Multi-National  Experiments 

-  Capt  Kathy  Shield,  Operational 
Experimentation  Branch  Head, 
Allied  Command  Transformation 
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EXPERIMENTATION  WORKSHOP 

Background:  Workshop  Overarching  Objectives 
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•  Overarching  issues: 

-What  is  analytical  rigor?  What  is  the  level? 

-Can  the  techniques  discussed  from  the  1999  MORS 
Workshop  on  Joint  Experimentation  be  used?  (specifically 
M-E-M) 

-Should  the  DoD  Command  and  Control  Research 
Program’s  (CCRPs)  Codes  of  Best  Practices  be  adopted 
as  an  analytical  best  practice? 

•  With  the  UK  and  US  leading  the  way,  the  “NATO  Code  of  Best  Practice  for 
C2  Assessment”  has  been  formally  adopted  by  several  governments  and  is 
having  a  positive  effect  on  the  quality  of  work  within  the  alliance. 

-What  is  the  extent  of  collaboration  in  experimentation? 

•  Among  the  Services?  Within  the  Coalition  Forces?  With  the  U.S. 
Agencies? 


EXPERIMENTATION  WORKSHOP 

Changes:  1999  to  2006 
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•  1999  Workshop  on  Joint  Experimentation 

-  To  develop  an  experimentation  process 

•  Key  Changes:  1999  to  2006 

-  Focus  shift  from  concepts  to  capabilities 

-  Now  at  war: 

•  Nature  of  warfare  from  MCO  to  GWOT 

•  Focus  changed  from  mid-term  to  near-term  emphasis 

•  Difficulty  bringing  a  team  together  for  large  events 

-  Better  tools  for  Model  -  Experiment  -  Model 

-  DODCCRP’s  Code  of  Best  Practices  for  Experimentation 

•  2006  Workshop  on  Bringing  Analytical  Rigor  to 
Joint  Warfighting  Experimentation 

-  To  determine  how  to  apply  analytical  rigor  across  the  process 
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EXPERIMENTATION  WORKSHOP 

Changes:  Enduring  Challenges 
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•  Enduring  challenges 

-  Lack  of  common  lexicon 

-  Planning  experimentation  campaigns  vice  single 
experiments 

-Joint  collaboration  -  international,  inter-agency, 
NGO/PVO 

-Tools  and  methods  to  support  efficient  &  effective 
experiments 

-  Educating  &  training  all  participants 
-Building  multi-disciplinary  teams 
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EXPERIMENTATION  WORKSHOP 

Overview:  Definition 
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•  Analytical  Rigor  in  Experimentation  - 

-  TOR: 

•  The  application  of  precise  and  exacting  standards  in  the 
examination  of  a  question  carried  out  under  controlled 
conditions  to  better  understand  and  draw  conclusions  in  order 
to  discover  an  unknown  effect,  to  test  a  hypothesis,  or 
demonstrate  a  known  fact,  and  uouoiiy  based  on  careful 
consideration  or  investigation. 

-  WG  Additions: 

•  WG  2:  “adherence  to  a  logical  process  of  addressing  issues 
through  metrics  and  measures  is  critical  to  achieving  rigor” 

•  WG  6:  “execution  of  a  well-defined  study  plan  with  appropriate 
parameter  space  that  has  subjective  and  empirical  results; 
traceable  to  data  that  supports  a  joint  warfighter  need.  ” 
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EXPERIMENTATION  WORKSHOP 

Summary:  Overarching  Objectives  Review 
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•  Overarching  issues: 

-What  is  analytical  rigor?  What  is  the  level? 

-Can  the  techniques  discussed  from  the  1999  MORS 
Workshop  on  Joint  Experimentation  be  used?  (specifically 
M-E-M) 

-Should  the  DoD  Command  and  Control  Research 
Program’s  (CCRPs)  Codes  of  Best  Practices  be  adopted 
as  an  analytical  best  practice? 

•  With  the  UK  and  US  leading  the  way,  the  “NATO  Code  of  Best  Practice  for 
C2  Assessment”  has  been  formally  adopted  by  several  governments  and  is 
having  a  positive  effect  on  the  quality  of  work  within  the  alliance. 

-What  is  the  extent  of  collaboration  in  experimentation? 

•  Among  the  Services?  Within  the  Coalition  Forces?  With  the  U.S. 
Agencies? 


EXPERIMENTATION  WORKSHOP 

Summary:  Overall 
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•  Providing  Analytical  Rigor 

-  Two  key  elements:  applying  the  Scientific  Method  &  providing  traceability  from  design 
to  data  to  results 

•  Using  the  Model-Experiment-Model  Paradigm 

-  With  limited  time  and  resources,  can  be  used  to  enhance  analytical  rigor 

•  Adopting  DODCCRP’s  Codes  of  Best  Practice  for  Experimentation 

-  Are  an  important  first  step,  provided  a  good  set  of  guidelines,  and  were  a  good 
foundation  for  experimentation 

•  Allowing  More  Consistent  Collaboration 

-  A  culture  change  is  necessary  to  foster  sharing,  create  DoD-wide  enforcement  or 
incentives,  develop  standard  terminology,  and  reuse  /  leverage  findings  across  the 
Services,  Multi-National  Forces,  Agencies  and  Non-Government  Organizations 


The  key  is  getting  the  analyst  involved  early  and  throughout  the  process. 
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EXPERIMENTATION  WORKSHOP 

Summary:  Why  Analytical  Rigor? 
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•  Why  have  analytic  rigor? 

-  More  confidence  that  the  insights  gained  are  not  misleading  decision 
makers 

•  Current  State  - 

-  Overall  -  Recognized  but  Inconsistent 

•  Key  elements  for  analytical  rigor: 

-  Use  the  scientific  method  and  proven  Operations  Analysis 
approaches  throughout  the  experiment 

-  Start  with  weli  defined  study  pian  with  appropriate  parameter  space 
that  has  subjective  and  empirical  results;  traceable  to  data 

-  One  of  the  items  that  many  of  the  working  groups  highlighted  was  the 
need  for  more  peer  review. 

•  The  synthesis  group  discussed  including  this  as  a  key  element,  but  decided  that 
this  was  an  integral  part  of  the  application  of  the  scientific  method  and  good 
operations  analysis  practice  and  decided  not  to  include  it  as  a  separate  element. 
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EXPERIMENTATION  WORKSHOP 

Summary:  COBP  as  a  Best  Practice 
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•  What  is  a  ^^Best  Practice?” 

-  “Best  practices”  that  have  been  found  useful  in  a  number  of  countries. 

-  They  have  been  widely  used  as  educational  tools,  guidelines,  stimuli  to 
ensure  quality  and,  in  some  cases,  roadmaps. 

-  They  are  not  rigid  rules  and  they  include  reasons  for  each  of  the  practices 
that  are  suggested. 

•  Previous  Experience 

-  With  the  UK  and  US  leading  the  way,  the  “NATO  Code  of  Best  Practice  for 
C2  Assessment”  has  been  formally  adopted  by  several  governments  and  is 
having  a  positive  effect  on  the  quality  of  work  within  the  alliance. 

•  Recommendation  to  MORS  Sponsors 

-  Adopt  DODCCRP’s  Codes  of  Best  Practice  for  Experimentation  as  a  MORS 
best  practice 

•  Decision  from  First  Sponsor’s  Brief 

-  Can  list  on  MORS  web-site  as  recommended  reading 


18 


EXPERIMENTATION  WORKSHOP 

Summary:  Model-Experiment  Model  (M-E-M) 
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•  1999  Workshop 

-  Recommended  to  structure  insights  and  provide  a  high  return  on 
investment  in  the  concept  exploration  phase. 

•  Discussion 

-  With  limited  time  and  resources,  a  model  could  be  used  to  shape  the 
experiment. 

-  During  the  experiment,  human-in-the-loop  information  could  be 
collected. 

-  This  information  could  be  modeled  for  further  analysis  after  the 
experiment 

•  Final  Thoughts 

-  M-E-M  could  be  used  to  structure  the  experiment  and  to  conduct 
follow-on  analyses. 
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EXPERIMENTATION  WORKSHOP 

Summary:  State  of  Collaboration  #  1 
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•  MORS  Current  Practice 

-  Annual  Symposium  to  share  work 

-  Special  Meetings  to  focus  on  priority  topics 

-  Very  little  follow-on  for  topics  and  meetings 

•  Some  exceptions:  CBP  and  Experimentation  Special  Meetings 

•  MORS  Leadership  Guidance 

-  MORS  President  Theme: 

•  Growing  (experimentation)  Analysts,  Expanding  Toolsets  (what’s  out  there  for 
experimentation),  and  Improving  Analysis 

-  MORS  President-Elect  Platform: 

•  “Launching  Change  for  the  Future”  (there’s  always  room  for  improvement) 

-  MORS  VP  (Meeting  Ops)  Advice: 

•  We  need  to  follow-up  on  Special  Meetings,  reporting  back  to  the  MORS  Sponsors 
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EXPERIMENTATION  WORKSHOP 

Summary:  State  of  Collaboration  #  2 
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•  Workshop  Recommendation 

-  Create  an  active,  collaborating  community  of  practice  of 
Experimentation 

-  Use  WG  33,  Warfighting  Experimentation,  as  an  active  peer  group 

•  Next  Steps 

-  Experimentation  Workshop  Chair  is  working  with  WG  33  Leadership 
to  hold  first  session  of  MORS  Experimentation  Community  of  Practice 

-  WG  Session  at  75^'^  MORSS 

•  Provide  overview  of  MORS  Experimentation  Workshop 

•  Query  attendees  on  what’s  been  done  since  the  workshop 

•  Establish  the  Experimentation  Community  of  practice  structure 

-  I.e.,  organization,  charter,  meeting  periodicity,  sub-groups 

•  Advertise  session  (e.g.,  Experimentation  Workshop  and  WG  33  database) 

-  After  the  MORSS,  report  to  MORS  Leadership  and  Sponsors  on 
what’s  been  done  and  the  next  steps 
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EXPERIMENTATION  WORKSHOP 

Summary:  JFCOM  /  J-9  Analytical  Challenges 
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•  New  methodologies  for  capabilities-based  experimentation.  How 
do  we  use  experimentation  to  evaluate  the  ability  of  potential 
solutions  to  meet  functional  capability  gaps? 

•  Integration  of  analysis  and  ^^meta-analysis”. 

-  How  can  we  “feed  forward”  knowledge  and  insights  from  one  event  to  the 
next? 

-  How  do  we  synthesize  significant  emerging  results  across  disparate  lines 
of  experimentation? 

-  How  can  we  build  the  body  of  knowledge  through  historical  and  thematic 
analysis  of  experimentation? 

•  More  comprehensive  and  rigorous  review  of  literature.  How  can 
we  develop  tools,  techniques  and  standards  to  more  effectively 
review  the  state  of  practice  and  the  academic  literature  and 
provide  valid  baselines  for  the  advancement  of  knowledge? 

•  Integration  of  established  qualitative  analytical  methodologies. 

-  How  can  we  incorporate  tested  methodologies  such  as  content-  or  factor- 
analysis  to  analyze  data  that  consists  largely  of  text? 

-  Can  such  approaches  be  considered  as  “rigorous”  as  traditional  OR 

techniques?  22 


EXPERIMENTATION  WORKSHOP 

Summary:  Recommendations  for  J-9  Challenges 
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•  Experimentation  in  Capabilities  Based  Assessments 

-  Use  existing  qualitative  analytical  methods  to  prioritize  /  filter  proposed 
solutions 

-  Use  Model-Experiment-Model  to  investigate  only  top  candidates 

•  Knowledge  Building 

-  Must  plan  and  resource  literature  review,  documentation,  archiving 

•  Require  in  deliverable,  integrate  into  schedule 

•  Emphasize  long-term  savings:  time  and  $$  (e.g.  reuse  of  data) 

-  Lead  by  example:  JFCOM  J9  adopt  internally 

-  Expand  to  services:  conduit  ->  JFCOM  J9  co-chair  MORS  Symposium 
Experimentation  Working  Group-33 

•  Qualitative  Analytical  Methodologies 

-  Considered  rigorous  if  used  appropriately 

-  Plan  for  traceability  throughout  the  experiment 

•  Multiple  replications  for  statistical  rigor  not  typically  available  in  experiments 
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EXPERIMENTATION  WORKSHOP 

Summary:  Working  Group  Discussions  #  1 
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•  WG  1:  Experiment  Design 

-  Issues  (from  their  perspective) 

•  Absence  of  peer  reviews 

•  Lack  of  a  knowledge  base 

-  Recommendations 

•  Create  an  active  advisory  group 

•  Improve  experimentation-related  definitions  &  terminology 

•  Need  to  demonstrate  the  value  and  relevance  of  Design  of  Experiments  methodology  in  terms  of  risk  reduction, 
profitability  and  efficiency 

•  WG  2:  Data  Collection,  Metrics  Evaluation  &  Reporting 

-  Findings 

•  The  Analysis  Plan  is  the  key  document;  it  is  a  contract.  If  not  followed,  results  may  not  be  guaranteed 

-  DO  NOTs 

•  Do  not  development  measurements  irrelevant  to  study  objectives 

•  Do  not  get  involved  too  late 

•  Do  not  simply  report  statistics  without  showing  operational  impact 
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EXPERIMENTATION  WORKSHOP 

Summary:  Working  Group  Discussions  #  2 
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•  WG  3:  Methodologies  and  Tools 

-  Shortfalls  (from  their  perspective) 

•  Lack  of  standards 

•  Lack  of  a  knowledge  base 

-  Recommendations 

•  Develop  experimentation-related  training 

•  Develop  a  reference  guide  for  methodologies  and  tools  across  the  experimentation  life  cycle 

•  WG  4:  People  as  Experimental  Assets 

-  Issues  (from  their  perspective) 

•  Inappropriately  using  Subject  Matter  Experts  (SMEs) 

•  Lack  of  relevant  expertise  in  human  behavior  analyses 

•  Lack  of  a  knowledge  base 

-  Findings 

•  During  the  planning  phase,  need  to  design  what  is  needed  from  SMEs 

•  Bias  from  SMEs  must  be  considered  and  managed  through  all  phases 
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EXPERIMENTATION  WORKSHOP 

Summary:  Working  Group  Discussions  #  3 
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•  WG  5:  Integration  &  Coordination  Across  DoD 

-  Issues  (from  their  perspective) 

•  Lack  of  shared  vision  on  Joint  Experimentation,  “coalition  of  the  willing” 

•  Cultural  mindset  is  needed 

-  Recommendations 

•  Encourage  information  sharing 

•  Create  a  knowledge  base 

•  Conduct  formal  peer  reviews 

•  WG  6:  Analytical  Rigor  to  Support  Capabilities  Based  Planning 

-  Issues  (from  their  perspective) 

•  Lack  of  common  definitions  and  terminology 

•  Lack  of  standards 

•  Lack  of  collaboration  and  access  to  other’s  data 

-  Recommendations 


•  Create  an  experimentation  lexicon 

•  Provide  experimentation  training 

•  Develop  a  knowledge  base 
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EXPERIMENTATION  WORKSHOP 

Status:  Documents 
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•  Status 

-  PHALANX  Article 

•  In  December  2006  issue 

-  Final  Report 

•  Review  completed  by  planning  committee  (Draft  and  Revision  1) 

•  Submitted  to  MORS  Communication  Manager  and  Publications 
Committee  on  12/14 

•  Being  reviewed  by  the  MORS  Sponsors  and  MORS  Office 
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EXPERIMENTATION  WORKSHOP 

Summary:  Final  Thoughts 
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•  Workshop  Results  /  Next  Steps 

-  Analytical  Rigor: 

•  Current  Level:  Inconsistent  Use  ^  Recommend  following  a  logical  process  with 
traceability 

-  M-E-M: 

•  Same  as  1999  ^  Propose  use  to  help  shape  experiments  and  conduct  follow-on 
analyses 

-  Best  Practice 

•  List  DODCCRP’s  Codes  of  Best  Practice  for  Experimentation  as  recommended 
reading  on  the  new  MORS  Experimentation  Community  of  Practice  web  pages 

-  state  of  Collaboration: 

•  Current  Level:  Infrequent  to  Inconsistent  Use 

•  Next  Steps  ^  Establish  an  active,  collaborating  Community  of  Practice  for 
Experimentation 

-  JFCOM/J-9  Challenges: 

•  Quick  response  by  attendees  to  Workshop  Proponent 
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EXPERIMENTATION  WORKSHOP 

Summary:  Remaining  Issues 
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•  Remaining  Issues  for  Experimentation  COP  to  Consider 

-  Developing  a  common  lexicon  for  experimentation  definitions  and 
terminology. 

-  Drafting  a  proposed  peer  review  process. 

-  Creating  a  list  (with  descriptions)  of  the  available  experimentation 
methodologies  and  tools. 

-  Developing  experimentation  training  materials. 

-  Investigating  the  requirements  and  how  to  collaborate  via  a  web-based 
approach. 

-  Ensuring  better  linkages  to  T&E,  Exercises,  JCIDs  /  JCAs. 

-  Developing  a  20-30  page  “How  To”  guide  (Sponsor  Recommendation). 

-  Providing  information  on  how  to  do  better  human-in-the-loop  experimentation. 

-  Establishing  experimentation  standards. 

-  Developing  a  list  of  common  experimentation  metrics 
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Experimentation 
Community  of  Practice  (COP) 
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Experimentation  COP 

Background:  Workshop  Collaboration  Review  #  1 
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•  MORS  Current  Practice 

-  Annual  Symposium  to  share  work 

-  Special  Meetings  to  focus  on  priority  topics 

-  Very  little  follow-on  for  topics  and  meetings 

•  Some  exceptions:  CBP  and  Experimentation  Special  Meetings 

•  MORS  Leadership  Guidance 

-  MORS  President  Theme: 

•  Growing  Analysts,  Expanding  Toolsets,  and  Improving  Analysis 

-  MORS  President-Elect  Platform: 

•  “Launching  Change  for  the  Future”  (there’s  always  room  for  improvement) 

-  MORS  VP  (Meeting  Ops)  Advice: 

•  We  need  to  follow-up  on  Special  Meetings,  reporting  back  to  the  MORS  Sponsors 
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Experimentation  COP 

Background:  Workshop  Collaboration  Review  #  2 
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•  Experimentation  Workshop  Recommendation 

-  Create  an  active,  collaborating  community  of  practice  of 
Experimentation 

-  Use  WG  33,  Warfighting  Experimentation,  as  an  active  peer  group 

•  Next  Steps 

-  Experimentation  Workshop  Chairs  and  WG  33  Leadership  held  first 
session  of  MORS  Experimentation  Community  of  Practice 

-  WG  Session  at  75*^^  MORSS  ^  THIS  SESSION 

•  Provide  overview  of  MORS  Experimentation  Workshop 

•  Query  attendees  on  what’s  been  done  since  the  workshop 

•  Establish  the  Experimentation  Community  of  practice  structure 

-  I.e.,  organization,  charter,  meeting  periodicity,  sub-groups 

-  After  the  75*^  MORSS,  report  to  MORS  Leadership  and  Sponsors  on 
what’s  been  done  and  the  next  steps 
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Experimentation  COP 

Organization:  Setting  up  a  COP 
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•  What  do  you  need  to  do  to  establish  a  Community 
of  Practice? 

V  Recommend  a  Community  of  Practice  area  (Experimentation) 

V  Develop  a  draft  charter  (developed) 

V  Create  an  organization  (have  leadership) 

V  List  some  potential  Issues  to  work  on  (have  list) 
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Experimentation  COP 

Organization:  Charter  and  Meetings 
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•  Experimentation  COP  Charter  (handout  with  details) 

-  Forum  for  communication  and  collaborating  within  MORS 

-  Foundation  from  2  MORS  Experimentation  Workshops 

-  Interface  with  experimentation-related  analysis  activities 

-  Methodology  developed  to  share  experimentation-related  info 

-  Experimentation-related  activities  identified  and  summarized 

•  Experimentation  COP  Meetings 

-  Annually,  as  part  of  WG  33,  Warfighting  Experimentation 

-  Regularly  throughout  the  year  via  MORS  telecon 

•  Full  COP:  Plan  is  to  meet  quarterly  as  a  COP 

•  Each  Sub-Group:  TBD,  perhaps  monthly  or  twice  a  quarter 
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Experimentation  COP 

Organization:  Experimentation  COP 
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•  Experimentation  organization 

—  Initial  Leadership  (Experimentation  Workshop  &  WG  33  Leadership) 

•  Chair:  Kirk  Michealson,  Experimentation  Workshop  Chair 

•  Co-Chairs: 

-  Brad  Baylor,  Experimentation  Workshop  Attendee  &  WG  33  Member 

-  Chris  Herstrom,  Experimentation  Workshop  Synthesis  Group  &  WG  33  Chair 

-  Dr.  Mike  McGinnis,  Experimentation  Workshop  WG  Chair  &  WG  33  Co-Chair 

-  Kemp  Littlefield,  Experimentation  Workshop  Attendee  &  WG  33  Advisor 

-  COP  Membership 

•  Analysts  volunteering  would  have  to  work  on  their  own  time  or  have  their 
organizations  support  it 

•  Why  should  analysts  work  in  this  new  Experimentation  COP? 

-  Build  up  their  experimentation  network 

-  Kept  informed  on  experimentation  issues  throughout  the  community  (could  be  common) 

-  Potential  professional  growth  opportunities  within  the  experimentation  community 
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Experimentation  COP 

Organization:  Goals 
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•  Experimentation  COP  Goals 

-  The  goal  is  not  to  tackle  all  possible  tasks, 

-  But  to  prioritize  the  experimentation  analytical  issues,  and 

-  To  select  a  few  where  the  Experimentation  COP  can  make  an 
immediate  difference. 

•  A  Sub-Group  is  planned  to  be  formed  for  each  issue 
being  worked 
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Experimentation  COP 

Next  Steps:  Remaining  Workshop  Issues 
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•  Issues  under  consideration  (not  in  priority  order) 

-  Developing  a  process  to  nominate  an  experimentation  analytical  issue. 

-  Developing  a  common  lexicon  for  experimentation  definitions  and  terminology. 

-  Investigating  the  requirements  and  how  to  collaborate  via  a  web-based  approach. 

-  Drafting  a  proposed  peer  review  process. 

-  Developing  an  Experimentation  COP  directory  with  names  and  contact  information  for 
members,  as  well  as  key  experimentation  personnel  and  organizations. 

-  Ensuring  there  are  instructions  on  the  Experimentation  COP  web  pages  on  the  MORS  web-site 
on  how  to  get  to  the  JFCOM  /  J9E  Joint  Experimentation  Knowledge  Portal. 

-  Creating  a  list  (with  descriptions)  of  the  available  experimentation  methodologies  and  tools. 

-  Developing  experimentation  training  materials. 

-  Ensuring  better  linkages  to  T&E,  Exercises,  JCIDs  /  JCAs. 

-  Developing  a  20-30  page  “How  To”  guide  (Sponsor  Recommendation). 

-  Providing  information  on  how  to  do  better  human-in-the-loop  experimentation. 

-  Establishing  experimentation  standards. 

-  Developing  a  list  of  common  experimentation  metrics. 

Which  are  the  right  ones  to  initially  work? 
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Experimentation  COP 

Next  Steps:  Upcoming  MORS  Events 
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•  Analysis  Support  for  Wargaming  Special  Meeting 

-  16-18  October  2007 

-  Heritage  Conference  Center,  NG-TASC,  Chantilly,  VA 

-  Ted  Smyth,  FS,  Chair 

-  Planning  Session,  0700  Thursday,  June  14*^,  Chauvenet  103 

•  Potential  Special  Meeting 

-  Joint  with  ITEA,  potentially  in  March 

-  Joint  Test  and  Evaluation  with  constructive  analyses 

-  Location:  TBD  (?  NM  ?) 
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Experimentation  COP 

Next  Steps:  Plans 
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•Next  Full  COP  Meeting 

-MORS  Telecon  in  September 

•  By  Next  Meeting 

-Form  Sub-Groups 

-Hold  Kick-Off  Meetings  for  Each  Sub-Group 

-Develop  initial,  draft  plan  of  action  & 
milestones  for  each  Sub-Group 
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Experimentation  COP 

Next  Steps:  Areas  and  Volunteers 
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•  Any  volunteers? 

-Experimentation  COP  Committee  Member 
-Experimentation  COP  Sub-Group  Lead 
-Experimentation  COP  Sub-Group  Member 
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Experimentation  COP 

Next  Steps:  Remaining  Workshop  Issues 
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•  Issues  under  consideration  (not  in  priority  order) 

-  Developing  a  process  to  nominate  an  experimentation  analytical  issue. 

-  Developing  a  common  lexicon  for  experimentation  definitions  and  terminology. 

-  Investigating  the  requirements  and  how  to  collaborate  via  a  web-based  approach. 

-  Drafting  a  proposed  peer  review  process. 

-  Developing  an  Experimentation  COP  directory  with  names  and  contact  information  for 
members,  as  well  as  key  experimentation  personnel  and  organizations. 

-  Ensuring  there  are  instructions  on  the  Experimentation  COP  web  pages  on  the  MORS  web-site 
on  how  to  get  to  the  JFCOM  /  J9E  Joint  Experimentation  Knowledge  Portal. 

-  Creating  a  list  (with  descriptions)  of  the  available  experimentation  methodologies  and  tools. 

-  Developing  experimentation  training  materials. 

-  Ensuring  better  linkages  to  T&E,  Exercises,  JCIDs  /  JCAs. 

-  Developing  a  20-30  page  “How  To”  guide  (Sponsor  Recommendation). 

-  Providing  information  on  how  to  do  better  human-in-the-loop  experimentation. 

-  Establishing  experimentation  standards. 

-  Developing  a  list  of  common  experimentation  metrics. 

Which  are  the  right  ones  to  initially  work? 
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WG  33  Name  Change 
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Experimentation  COP 

WG  33:  Potential  Name  Change 


Military  Operations  Research  Society 


•We  are  doing  a  lot  more  than  Warfighting 
Experimentation 

-  Global  War  of  Terror 

-Conflict  resolution,  stability  operations 

-  Emergency  response  and  emergency  management 

-  Current  Operations  -  order,  security  and  stability 

-  Integrating  US  Forces  with  host  nations 

•  What’s  the  right  working  group  name? 

-Any  suggestions? 
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EXPERIMENTATION  WORKSHOP 

Status:  Recommendations 
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*  Recommendations 

-  From  the  Workshop: 

•  Adopt  DODCCRP’s  Codes  of  Best  Practice  for  Experimentation  as  a  MORS  best  practice 

•  Use  WG  33,  Warfighting  Experimentation,  as  an  active  community  of  practice  and 
advisory  group 

•  Create  a  community-wide  database  for  experimentation  (JFCOM  /  J-9  may  be  a  logical 
choice) 

-  Future  Tutorials: 

•  How  to  conduct  a  literature  search 

•  Mid-level  tutorial  on  Design  of  Experiments 

-  Future  Special  Meetings: 

•  Experimentation  Workshop  every  2  years 

•  Limited  Objective  &  ACTD  Experimentation 

•  Discovery  &  Human-in-the-Loop  Experiments 

•  Training  to  provide  analytical  rigor  in  Joint  Warfighting  Experimentation 

•  Application  of  experimentation  results 
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